Modeling Second Language Acquisition with pre-trained neural language models

Prediction of language mistakes is a task introduced by Duolingo as part the Second Language Acquisition Modeling topic that aims to learn from history improve experience learners. Using transfer learning means pre-trained models, we propose framework can actual distribution according which faraway words sentence have higher chance produce errors. To adapt information provided more approaches based on feature extraction or fine-tuning were tried. However, our experiments, integrating these two options in stack-and-finetune approach seems be appropriate for task. Regarding comparison models terms model distillation, notice distillation does not affect effectiveness while significantly reducing training time. We conclude complexity should adjusted specifics analyzed problem and an efficient option low corpora without considerably affecting overall performance.